Skip to content

Remote logging fix#68370

Merged
jason810496 merged 2 commits into
apache:mainfrom
m8719-github:remote-logging-fix
Jun 16, 2026
Merged

Remote logging fix#68370
jason810496 merged 2 commits into
apache:mainfrom
m8719-github:remote-logging-fix

Conversation

@m8719-github

@m8719-github m8719-github commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

closes: #68366
related discussion on Slack: https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1780575542424439

Do not cache None connection values at the start of the worker lifecycle so that the worker is able to upload task logs upon task completion. Please see linked issue for full details.

Tested via monkey patching on a test environment, logs showed up in the UI and in S3 with this change.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)


Important

🛠️ Maintainer triage note for @m8719-github · by @potiuk · 2026-06-12 11:21 UTC

Some review feedback from @jason810496 is waiting on you:

  • 1 unresolved review thread(s) on this PR.

The ball is in your court — you've been assigned to this PR. Reply or push a fix in each thread, then mark them resolved (and ping the reviewer when you're ready).

Automated triage — may be imperfect; a maintainer takes the next look.

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! The patch make sense to me overall.
Would appreciate to add corresponding unit test when you have a moment, thanks.

Comment thread task-sdk/src/airflow/sdk/execution_time/supervisor.py
@ashb ashb marked this pull request as ready for review June 11, 2026 11:02
@ashb ashb requested review from amoghrajesh and kaxil as code owners June 11, 2026 11:02
@ashb ashb added the type:bug-fix Changelog: Bug Fixes label Jun 11, 2026
@ashb ashb added this to the Airflow 3.3.0 milestone Jun 11, 2026
@ashb

ashb commented Jun 11, 2026

Copy link
Copy Markdown
Member

However the entire pre-fetch might be moot now. This is caching to a single supervisor process which shuts down when the task finishes. There's not much point in trying to pre-cache as it will always fail now?

However this leads to an interesting problem -- some loggers such as GCP's StackDriver or AWS's Cloudwatch Logs want to upload logs as they come in, not in a batch at the end, and for those to work we need to know that we weren't able to get logs and try again once the task has started running and we can access connections.

(Which isn't to say we shouldn't have a tactical fix like this to help most loggers, but the main issue is not resolved until we fix it for all)

@m8719-github

Copy link
Copy Markdown
Contributor Author

However the entire pre-fetch might be moot now. This is caching to a single supervisor process which shuts down when the task finishes. There's not much point in trying to pre-cache as it will always fail now?

However this leads to an interesting problem -- some loggers such as GCP's StackDriver or AWS's Cloudwatch Logs want to upload logs as they come in, not in a batch at the end, and for those to work we need to know that we weren't able to get logs and try again once the task has started running and we can access connections.

(Which isn't to say we shouldn't have a tactical fix like this to help most loggers, but the main issue is not resolved until we fix it for all)

Yes, this occurred to me as well but I wanted to be as tactical with the fix as possible. I'll take a closer look at GCP StackDriver and AWS Cloudwatch remote logging sub-systems in a separate branch and see what can be done there to fix this issue.

@m8719-github

Copy link
Copy Markdown
Contributor Author

Nice! The patch make sense to me overall. Would appreciate to add corresponding unit test when you have a moment, thanks.

yep, will add one before marking ready for review; should be done in a day or two.

anleib and others added 2 commits June 14, 2026 13:24
Signed-off-by: AndreiLeib <andrei.leibovski@appdirect.com>
Signed-off-by: AndreiLeib <m8719-github@yahoo.com>
@jason810496 jason810496 merged commit 41f20d5 into apache:main Jun 16, 2026
102 checks passed
RulerChen pushed a commit to RulerChen/airflow that referenced this pull request Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:task-sdk type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote logging not working in Airflow 3.2.2

4 participants